Skip to content

Add eFP/ePlant gene ID validation, microarray probeset support, and m…#324

Open
rmobmina wants to merge 12 commits into
BioAnalyticResource:devfrom
rmobmina:cleaned-endpoint
Open

Add eFP/ePlant gene ID validation, microarray probeset support, and m…#324
rmobmina wants to merge 12 commits into
BioAnalyticResource:devfrom
rmobmina:cleaned-endpoint

Conversation

@rmobmina

Copy link
Copy Markdown
Contributor

No description provided.

rmobmina added 12 commits June 25, 2026 11:11
…aster DB list

Adds per-eFP-project regex validation (EFP_PROJECT_REGEXES, is_efp_gene_valid)
covering both canonical gene IDs and microarray probeset IDs, with a
database-to-project mapping (DATABASE_EFP_PROJECT) so gene_expression.py can
validate against the right format per database.

Adds the eFP+ePlant discovery/validation pipeline:
- scrape_view_databases.py / scrape_species_view_info.py: live-discover every
  eFP and ePlant view across 55 sites, tagging eFP-only vs ePlant-only vs both
- build_proj_id_view_mapping.py: resolves multi-paper databases (e.g.
  atgenexp_*) to per-view proj_id breakdowns
- validate_db_regex_coverage.py: tests every database's real sample IDs
  against production validation, including legacy databases no longer linked
  from a live dropdown
- build_master_db_list.py: rolls everything up by species, tagging source
  (efp/eplant/both/legacy) and platform (microarray/rna_seq)

193 databases validated, 0 unexplained failures.
… endpoints to use it

- Merge heterodera_schachtii into arabidopsis (it's an Arabidopsis-gene-ID
  database, not a separate species); fix brassica/lupin/triticale scientific
  names per professor's review comments
- Load EFP_PROJECT_REGEXES and DATABASE_EFP_PROJECT from Vincent's
  regex_master_list_efp_eplant registry (58 grouped project patterns instead
  of 193 per-database ones), embedded into combined_master.json at build
  time; empirically verify every regex_project assignment against real
  sample dumps before trusting it, demoting 3 unverified species-fallback
  guesses and fixing 2 misassigned projects (maize_lipid_map, tomato_trait)
- Add a general injection checker (BARUtils.is_injection_attempt) that runs
  before the probeset-shape check in /gene_expression, closing a bypass
  where any string ending in "_at" skipped format validation entirely
- Derive schema_variants directly from real sample_data column dumps instead
  of a coarse rna_seq/microarray platform split, so every one of the 193
  databases is schema_verified against its own real columns (was 146/193)
- Refactor /microarray_gene_expression/<species>/databases and
  .../<species>/<view>/samples to read from combined_master.json instead of
  a hardcoded dict and a since-deleted JSON file (the latter was broken on
  this branch); add test coverage for both
- Add api/utils/master_data_utils.py as the shared combined_master.json
  loader used across bar_utils, gene_id_utils, and the refactored resource
These were leftover exploratory outputs (regex audit dumps, coverage CSVs,
speed benchmark plots, old email drafts) not consumed by any script in the
active build pipeline. db_regex_coverage_report.csv is kept since
build_master_db_list.py reads it as an input.
These "Reena Obmina | ... Project | University of Toronto" docstring
lines and "# Endpoint made by Reena" comments were course-assignment
artifacts, not something that belongs in a shared codebase.
build_databases() collected each database's used_by entries from a
Python set, so re-running the exact same build against the exact same
inputs produced a different (but equivalent) entry order every time,
due to Python's per-process string hash randomization. Sort before
iterating so reruns are byte-identical -- important since this file is
meant to be diffable for review, not just functionally correct.

Regenerated data/efp_info/combined_master.json with the fix; verified
two consecutive reruns now produce byte-identical output, and the only
diff from the previous version is list ordering (no data changed).
- test_interactions.py::test_mfinder special-cased GITHUB_ACTIONS to
  skip the real /bartmp/mfinder subprocess call, but CI doesn't
  actually have that binary installed either -- generalize the check
  to "skip whenever the binary isn't present" so it behaves the same
  in CI, locally, or anywhere else without mfinder installed.

- Delete tests/resources/test_eplant_arabidopsis.py: it tests
  api.resources.expression.get_expression and the route
  /expression/ePlant_expression, neither of which exist anymore --
  that module was renamed/replaced by the unified
  /expression/<database>/<gene_id> endpoint in gene_expression.py,
  which already has its own Arabidopsis coverage in
  test_gene_expression.py. No coverage lost.
These 8 import_if_missing lines (light_series, maize_RMA_linear,
meristem_db, potato_stress, seedcoat, soybean, soybean_severin,
tomato_ils) could never actually run: bootstrap_simple_efp_dbs.py
runs earlier in the same script and already creates all of these as
empty databases from the schema registry, so db_exists() always
returns true and the import is skipped before it ever reaches the
SQL file. Most of those files are also gitignored multi-GB dumps
that don't exist in a checkout anyway (soybean.sql doesn't exist at
all). Removing dead lines that can never execute rather than leaving
confusing no-op code in a script CI actually runs.
Walks every eFP browser view for every species in
data/efp_info/combined_master.json and checks whether each
"To the Experiment" hyperlink shown in the expression-value table
actually resolves to something real, instead of erroring out or
looping back into the eFP browser itself.

Ignore its scratch output directory (scripts/results/) rather than
letting the generated CSV/log get committed by accident.
# Conflicts:
#	api/utils/bar_utils.py
#	config/init.sh
BARUtils had 46 hand-maintained is_<species>_gene_valid() methods that
duplicated what Vincent's tested regex registry (EFP_PROJECT_REGEXES,
loaded from combined_master.json) already covers per eFP project --
exactly the kind of drift-prone duplication that caused the sorghum
regex merge conflict with upstream/dev. Delegate 41 of them to
is_efp_gene_valid() so there's one tested source of truth instead of
two. The remaining 5 (cassava, cuscuta, spruce, willow, quinoa) have
no matching project in Vincent's registry and stay hand-maintained.

Verified every migrated species against every real sample ID in
api/random_rows_json/ before switching: 26 were byte-identical, no
behavior change. The other 15 (including cacao's 3-way OR and mouse's
differently-named mouse_efp key) do change behavior, knowingly:
- Most now also accept microarray probeset IDs (e.g. 253680_at,
  Contig7905_at) as "valid genes" for that species, since Vincent's
  registry validates "any real value in this database's sample_data"
  rather than "a canonical gene ID a human would type."
- Rice loses isoform-suffix support entirely (LOC_Os01g01430.1):
  efp_rice has no isoform variant of the LOC_Os form. This affects
  /loc/rice/<gene>'s real primary use case, not just an edge case.
- Wheat, canola, and potato lose a few legacy real-world formats
  (e.g. TrturSVE..., BrBA_...V4, EPlSTUG...) that efp_wheat/
  efp_canola/efp_potato don't cover.
- Arabidopsis gains a two-digit isoform suffix (At1g01010.11) and
  loses a previously-tolerated malformed form (AT1G010400, an extra
  trailing digit with no separator).

Also fixes is_efp_gene_valid() itself: it was matching without
re.IGNORECASE while every legacy method it's now replacing used
re.I, which broke case handling for callers like efp_data.py that
uppercase the gene ID before validating (e.g. rejected a real,
valid Arachis ID). And added a None/empty guard so callers that
pass an absent gene id don't crash instead of failing validation.

Updated test_bar_utils.py, test_gene_information.py, and
test_localizations.py to match the new, intentional behavior.
config/init.sh ran bootstrap_simple_efp_dbs.py (creates an empty
sample_data table for all 191 "simple" eFP databases) BEFORE the
static SQL imports. Several databases -- shoot_apex, arachis,
single_cell, arabidopsis_ecotypes, cannabis, dna_damage, embryo --
are both a "simple" schema AND have a small curated real-data dump in
config/databases/. Since bootstrap always created their table first,
db_exists() saw it as "already there" and import_if_missing skipped
loading the real row every time, leaving these tables permanently
empty in any genuinely fresh environment (CI, a new dev machine).
This only looked fine locally because that database already had the
real data loaded from long before the bootstrap step existed.

Fixed by running the static imports first (so a fresh database gets
its real data loaded), then bootstrapping the rest -- safe because
ensure_schema() uses checkfirst=True and never touches a table that
already exists. Also added cannabis, dna_damage, and embryo to the
import list; their real dumps existed in config/databases/ but were
never referenced anywhere, going all the way back to before the
import_if_missing refactor.

Verified locally: dropped these three databases entirely, replayed
the corrected sequence (real-data insert, then a second bootstrap
pass), and confirmed the real row survives and the dependent tests
pass with it.

Also pinned rpds-py to 0.30.0 (was 2026.5.1, from upstream/dev's
"Dependancies updated." commit): 2026.5.1 requires Python >=3.11 and
has no compatible release for the 3.10 job in the CI matrix, which
made `pip install -r requirements.txt` fail outright on that job
before tests could even run. 0.30.0 is the newest version with a
3.10 wheel and has wheels for 3.11/3.12/3.13 too.
193 files, ~52k lines of raw per-database sample dumps -- used only as
input to build_combined_master_json.py's schema/regex verification
against real data, never read at runtime by the API or by any test.
Too large and noisy to be worth reviewing line-by-line in a PR.
Gitignored (not deleted) so the build script still works locally for
whoever has a copy; regenerate combined_master.json before removing
locally without a backup.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant